Eye-gaze tracking

ABSTRACT

A device and a method for tracking an eye-gaze of an observer. A deep blue or violet light source is used to emit light to eye, particularly to the retina. The deep blue light is partially reflected and partially absorbed by the retina. The absorption is most prominent around the fovea, the area of sharp vision, because of the pigment which protects the fovea from short wavelength radiation. Thus the device and method of tracking eye-gaze according to the invention comprises emitting light having a certain wavelength and transferring the light to the retina of an eye. The wavelength of the light being such as to make the fovea of the eye resolvable. The method further comprises detecting light that is reflected from the eye to form detection information including the resolvable fovea, and mapping the detection information to a predetermined surface, the surface being located at a distance from the eye, the location of the fovea on the surface forming an eye-gaze point.

FIELD OF THE INVENTION

[0001] This invention relates to eye-gaze tracking and, moreparticularly to an eye-gaze tracking device and method preferably fortracking the eye-gaze of a user on a surface such as a display.

BACKGROUND OF THE INVENTION

[0002] Keyboards, mice or joysticks provide a communication interfacebetween a human and an electronic device. Eye-gaze measurements havebeen used in physiological and psychological studies disclosed indocument Arne John Glenstrup and Theo Engell-Nielsen: BS thesis,Laboratory of Psychology, University of Copenhagen which is available inInternet at the URL http://www.diku.dk/˜panic/eyegaze/article.html.Further eye-gaze tells about an interest area of an observer.

[0003] Present eye trackers are based on reflections of Infra Red (IR)light from the different layers of the eye, which is known as Purkinjereflections, and on reflection from a retina, which is seen as a brightiris. There are trackers that use only Purkinje images and trackers thatcombine the reflections from cornea and bright iris. Those reflectionsmove with respect to each other and with respect to the bright irisdepending on gaze direction. Usually one IR point source is enough butin order to increase accuracy several IR sources have been used. Theretina has very large reflection at red and even more at IR wavelengths.Also the visibility of details, e.g. blood vessels, vanish when usinglonger wavelengths. Thus the retina looks very uniformly illuminated.This is why the iris-cornea method uses IR wavelengths.

[0004] Present eye trackers faces major drawbacks to be able to fulfillthe need for general usage. Eye trackers that are based on the Purkinjeand/or the so called iris-cornea method must be calibrated frequentlyand a calibration is user dependent. In addition of that, these devicesare not suitable for all people, because the eye structure of somepeople is incompatible for the device. This is because of physiologicaldifferences in the eye. Especially physiological differences in eyelidpositions. Other concerns are slowness of the device, that is a delay inthe existing devices prevents the required control.

[0005] An additional drawback is a general eye tracking structure ofpresent eye trackers. Present eye trackers are bulky and heavy on asystem level. These eye trackers require systems which are too massivefor intelligent integrated electronic devices such as displays, virtualreality computer displays, portal computers and mobile phones.

[0006] Present eye-gaze tracking methods monitor the outer parts of aneye. These methods are very much person dependent and they are too muchaffected by personal eye geometry or the position of eyelids. Thuspersonal differences of eye physiology sets a restriction for a commonusage. These methods are inaccurate for required controlling andtracking.

[0007] A need for an improved user interface setting hands free isevident, as the devices become smaller, portable, more intelligent andubiquitous. Eye-gaze tracking should set a data stream delivering a highinformation content to the part of display eye is gazing. Efficientmonitoring of eye-gaze is inevitable to eye-gaze controlledcommunication between a human and an electronic device. A control setshould be eye-gaze itself or eye-gaze combined to other existing controlset such as a button. In addition, an eye-gaze tracking/control deviceshould provide a response with an imperceptible delay for a device userfor required usage. Current devices do not meet these abilities or theyprovide infeasible implementations.

SUMMARY OF THE INVENTION

[0008] The present invention provides a device and a method for trackingan eye-gaze on a surface such as a display of an electronic device andthus provides a base for controlling the electronic device according tothe eye-gaze of a user.

[0009] According to a first aspect of the invention there is provided amethod of tracking eye-gaze, the method comprising

[0010] emitting light having a certain wavelength;

[0011] transferring the light to the retina of an eye;

[0012] the wavelength of the light being such as to make the fovea ofthe eye resolvable;

[0013] detecting light that is reflected from said eye to form detectioninformation including the resolvable fovea;

[0014] mapping the detection information to a predetermined surface,said surface being located at a distance from said eye, the location ofthe fovea on said surface forming an eye-gaze point.

[0015] According to a second aspect of the invention there is providedan eye-gaze tracking device, comprising:

[0016] a light source for emitting light having a certain wavelength;

[0017] means for transfering the light to the retina of an eye, thewavelength of the light being such as to make the fovea of the eyeresolvable;

[0018] a detector for detecting light that is reflected from said eye toform detection information including the resolvable fovea;

[0019] a surface located at a distance from said eye; and

[0020] means for mapping the detection information to the surface forlocating the the fovea on said surface for forming an eye-gaze point bythe location of the fovea.

[0021] In a preferred embodiment of the invention a deep blue or morepreferably violet light source is used to emit light into the eye,particularly to the eye retina. Accordingly this means light having awavelength in the range of about 395-500 nm, where a violet light havinga wavelength of about 395-430 nm is preferred but blue light upto awavelength of about 480 nm also works. Further an optical x-y matrixdetector is used to measure the light reflected from eye retina. Fromthe reflected light the foveal position, which is the eye-gaze, withrespect to the optical assembly is measured with the detector.

[0022] According to another embodiment of the invention an eye-gazetracker is integrated to a a display, which can be a virtual realitydisplay.

[0023] In one embodiment of the invention, eye-gaze control can set adata stream delivering a high information content to the part of displaythat the eye is gazing. Also in an embodiment a control set or a controlmaster is eye-gaze combined to a hand controlling equipment. Accordingto an embodiment of the invention there may be provided a software whichmeasures and calculates fovea from data received from the detector,preferably the x-y detector, which may be a CCD (Charge Coupled Device)for example. In a further embodiment, the invention is provided with apattern recognition calculation of blood vessels and comparing thecalculations with a calibration pattern of blood vessels.

[0024] The invention provides several advantages over prior solutions.For example, a device according to the method of the invention may beimplemented to be simple and light eye-gaze tracking device. Thisenables an embodiment for Virtual Reality (VR) displays having areasonable physical size. Moreover, for portable computers or mobileunits, such as mobile phones, a device for eye-gaze tracking accordingto the invention of reasonable structure can be implemented.

[0025] Preferably tracking and detecting the eye-gaze is based on theobservation of the retina. This follows that the invention functions fordifferent eyes although personal physical eye structure may varyconsiderably. This allows a large amount of users to apply theinvention. The invention is thus suitable for substantially all usershaving normal physical human eye structure.

[0026] Moreover, the observation of the retina gives reliable andprecise information of eye-gaze. Eye-gaze direction can preferably bedetermined without calibration of the device. Thus it is possible toobtain eye-gaze position on a display within immediate reaction from theeye-gaze device having measured with the detector (such as a CCDdetector) and mapped to a display, e.g. a LCD.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The disclosed invention will be described with reference to theaccompanying drawings, which show embodiments of the invention andwhere:

[0028]FIG. 1 depicts a block diagram of the device architecture of apreferred embodiment,

[0029]FIGS. 2a and 2 b depict an elucidation for utilization aboutphysical feature of retina of healthy eye at IR and at blue/violetwavelengths,

[0030]FIG. 3 depicts a flow chart of a preferred embodiment of acalibration free eye-gaze tracking,

[0031]FIG. 4 depicts a flow chart of another embodiment of the eye-gazetracking, and

[0032]FIG. 5 depicts a typical use situation of the eye tracking deviceof the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The invention will be described with particular reference to apreferred embodiment. However, it should be understood that thedifferent embodiments provide only a few examples of the manyadvantageous uses of the innovative teachings herein.

[0034] Various embodiments of the disclosed method and device will bedescribed utilizing the physical feature of the retina. FIGS. 2a and 2 bdepict an elucidation for utilization about the physical feature of aretina of a healthy eye at IR (infrared) light and at blue light orviolet light wavelengths. In FIG. 2a shows the retina of a healthy eyeilluminated at IR light wavelengths. The retina has very largereflection at red and even more in IR wavelengths. Also the visibilityof details, e.g. blood vessels, vanish when using longer wavelengths. Onthe other hand when the wavelength is shortened, more and more detailscome out. Not only blood vessels but also fovea emerges. This can beseen from FIG. 2a where at IR light the fovea cannot be detected. FIG.2b shows the retina of a healthy eye illuminated at blue wavelengths. Atblue light the fovea emerges as a dark spot and the further we go toshorter wavelengths the more prominent the fovea is. Thus a deep blue orviolet light sets a clear and prominent fovea. This is due to the UVprotective pigment (xantofyllein-lutein), which is densest just in thefovea. The deep blue or violet light is partially reflected andpartially absorbed by the retina. The absorption is most prominentaround the fovea, the area of sharp vision, because of the pigment whichprotecs the fovea from short wavelength radiation. This yellow pigmentedarea is called as macula lutea. Because of the absorption (whereas otherparts of the eye reflect light at this wavelength) the fovea becomesresolvable as a dark spot. Accordingly this means light having awavelength in the range of about 395-500 nm. A violet light having awavelength of about 395-430 nm is preferred. When a light of longerwavelength is used the fovea does not anymore emerge very clear. Thefovea has been detected by blue light having a wavelength of about 480nm. Thus even a wavelength of 500 nm and a bit above still brings outthe fovea but when moving closer to red light (600-700 nm) the fovea canno longer be detected as shown in FIG. 2a. The fovea does appear clearlyalso for light having a wavelength less than 395 nm, but that is notrecommended as it may damage the eye. Thereby light with a wavelength of395-430 nm is preferred, and good results have been achieved with 405nm.

[0035]FIG. 1 depicts a block diagram of the device architecture of apreferred embodiment of the invention. There an eye-gaze tracking deviceis integrated to a Virtual Reality Display (VRD). The VRD comprisesoptical parts such as a Liquid Crystal Display (LCD) panel 108, imagingoptics 104, illumination optics 114 and a polarizing beam splitter 110.The illumination optics has a light source 112 to illuminate the LCDpanel 108. The prisms 124 and 126 and illumination optics 114 are formedso that the LCD panel 108 has homogeneous illumination and also so thatthe amount of light entering the eye is a safe amount of light in ordernot to cause physical harm or damage to the eye. The imaging optics maycontain lenses, mirrors and alternatively even diffraction optics.

[0036] The eye tracking optics comprises a narrow band 118 beam splitter(implemented as a coating), a deep blue or violet source 116 forillumination of the retina 100. In two passes through the quarter waveplate 122 (which also is narrow band, e.g. in the range of 405 nm), thepolarization of the short wavelength light will rotate by 90 degree butdoes not affect to the other bands of spectrum (except the band of thequarter wave plate). The LCD panel 108 does not alter the polarizationstate of this short wavelength light. The illumination optics 114,polarizer 120, prisms 124 and 126 and the quarter wave plate 122 work sothat the short wavelength light is guided to the retina 100. The shortwavelength light, which is reflected from the retina, is going back tothe device, but now the narrow band reflector 118 guides the lighttowards the detector 106. The focal plane of this light is inside theprism 126, and a relay lens 128 transforms the image from this planeonto the detector 106. The relay lens also takes care of correctscaling, because the display panel and the detector can have differentdimensions. The detector 106 can be for example a CCD camera and is usedto convert optical information to electronic or binary information. TheLCD panel 108 is a reflective microdisplay and is used as a traditionaldisplaying unit and as a function unit to set a location to eye-gaze.The display panel 108 can be also transmissive but in this case theillumination is assembled behind the panel. The light source can beimplemented as a LED (Light Emitting Diode) and also the other opticsshown in FIG. 1 (i.e. the parts except the display) can be made of smallsize and thus this may made in a size such as 18 mm×18 mm×6 mm.

[0037] In the embodiment of FIG. 1, the imaging optics 104 transformsand transfers a picture of the LCD panel to a picture on the retina 100.This is done to achieve a clear picture on the retina 100. Thus theimaging optics 104 and also the lens of the eye 102 sets rays from theLCD panel 108 to the retina 100 in such a way that the picture on theretina 100 is visible and noticeable according to its source from theLCD panel 108. The illumination optics 114 is formed so that the LCDpanel 108 has a homogeneous illumination and also so that the amount ofshort wavelength light 116 entering the eye is optimized, that is a safeamount of light enters eye that causes no physical harm or damage to theeye. The short wavelength of light (˜400 nm or at least less than 500nm) can be so low of intensity or so short pulses used, that the humaneye cannot resolve the light. Thus the eye tracking operation is notobserved by the user whereas the detector can be built very sensitive tothis part of light spectrum. Thus the light source 112 illuminates theLCD panel 108 homogeneously and has no considerable interference to thedetection at the detector 106 about light reflected from the retina 100to the detector 106. Also on the other hand, the short wavelength source116 has no considerable interference to rays forming picture emitted bythe LCD panel 108 to the retina 100. The deep blue or violetillumination is arranged so that it visible when the LCD panel 108 ishomogeneously illuminated by deep blue or violet light. The image ofretina is thus transformed on to the detector 106. There will be somestray light from the cornea, the iris, the pupil and the lens but theseparts are not in focus and therefore they contribute only to background,thus having no considerable interaction to an observer.

[0038] In the embodiment of FIG. 1, the aim is to measure a location ofa fovea on the detector 106. The fovea sets a one to one correspondenceto an eye-gaze on surface, which eye is gazed and a fovea spot isdetected and measured. A position of the fovea spot on the detector 106has one to one correspondence to a position on the LCD panel. Thusmeasuring on the detector 106 a fovea spot, which is eye-gaze, utilizingone to one correspondence between the detector 106 and the LCD panel108, a fovea spot on the LCD panel 108 is formed, which is now aneye-gaze on the LCD panel 108. The fovea spot appears as dark spot onthe picture of the reflected light from the retina 100. Thus an aim isto measure this dark spot. The detector 106 detects and converts adetection information to data, which is a numerical or electrical formof detection information. This data is transferred to a controller unit107. A running software in control unit 107 measures and calculates thefovea spot from the data and transfers this information to the deviceusing the display.

[0039] Alternatively, the location of a fovea may be obtained from CCDdata, where a CCD detector is used as detector 106. Detectioninformation and thus CCD data is obtained as described above. Also theentire VRD device can be corresponding as in the preferred embodiment.The location of a fovea from CCD data may be obtained by using acomparison of patterns. From CCD data an initial calibration pattern ofblood vessels is formed in control unit 107. After a calibration patternis set, a new measurement and thus a new CCD data of an eye-gazeposition is detected on the CCD detector 106. From new CCD data apattern of blood vessels is formed in the control unit 107. The controlunit performs a series of functions to implement a pattern recognitioncalculation of these two patterns (pattern recognition by comparing ameasured pattern with a calibration pattern being known as such). Thecontrol unit 107 performs a series of calculations to implementcomparison between calibration pattern and eye-gaze pattern. By thesecalculations and a comparison (of the measured and calibration pattern)the control unit 107 sets the location of a fovea spot on the CCDdetector 106. Mapping the location of a fovea, moreover eye-gaze,between the CCD detector 106 and the LCD panel 108 is implementedcorrespondingly as was described above, and yet by obtaining the foveafrom CCD data using pattern comparison.

[0040]FIG. 3 depicts a flow chart of an embodiment of the device andmethod of tracking eye-gaze according to the invention. A deep bluesource 116 emits a deep blue or violet light (Step 300). A narrow bandblue reflector 118 reflects the emitted light appropriately according tofrequency and direction (Step 302). The emitted and reflected lightreflects back from the eye retina (Step 304). Light reflected from theeye retina reflects from narrow band blue or violet beam splitter 118appropriately according to frequency and direction (Step 306). Adetector 106 detects the reflected light and converts this lightdetection information to electronic or binary data form (Step 308). Thedetector 106 measures a fovea spot from the data (Step 310). The controlunit 107 maps the fovea spot from the detector 106 surface (Step 312).The device implements functions depending on mapped eye-gaze on VRD,e.g. display eye-gaze on LCD panel thus on VRD or set control toeye-gaze location (Step 314).

[0041]FIG. 4 depicts a flow chart of an alternative embodiment of thedevice and method of tracking eye-gaze according to the invention. Adeep blue source 116 emits a deep blue or violet light (Step 400). Anarrow band blue reflector 118 reflects the emitted light appropriatelyaccording to frequency and direction (Step 402). The emitted andreflected light reflects back from the eye retina (Step 404). Lightreflected from the eye retina is further reflected from narrow band blueor violet beam splitter 118 appropriately according to frequency anddirection (Step 406). A detector 106 detects the light reflected fromthe narrow band mirror 118 and converts the light detection informationto electronic or binary data form (Step 408). The control unit 107calculates a pattern recognition calculation of blood vessels from thedata (Step 410). If an initial calibration pattern does not exists, thecalculated pattern is set to form an initial calibration pattern (Steps411 and 412). The device then sets a new CCD data from streamingeye-gaze information. If initial calibration pattern exists, the controlunit 107 compares the calculation to calibration pattern (Steps 411 and414). The control unit 107 maps the fovea spot from the CCD detector 106surface to the LCD panel 108, thus to VRD surface (Step 312). The deviceimplements functions depending on the mapped eye-gaze on VRD, e.g.display eye-gaze on the LCD panel thus on the VRD or sets control toeye-gaze location (Step 314).

[0042] Eye-gaze tracking/control should set a data stream delivering ahigh information content to the part of the display that the eye isgazing. Efficient monitoring of eye-gaze is inevitable to eye-gazecontrolled communication between a human and an electronic device. Acontrol set should be eye-gaze itself or eye-gaze combined to otherexisting control means such as a control button. In addition, aneye-gaze tracking/control device should provide a response with animperceptible delay for a device user for required usage.

[0043]FIG. 5 depicts an example of a typical use situation of theeye-gaze tracking and controlling device. In FIG. 5, a user or anobserver 500 utilizes the eye-gaze tracking device integrated into a VRDdevice 504. A user's eye 502 is used to control VRD device 504.

[0044] After eye-gaze is mapped to VRD, VRD device 504 is in standbymode to utilize eye-gaze based controlling to the device used. Eye-gazecontrol can set a data stream delivering a high information content tothe part of the display 504 that the eye 502 is gazing. Thus a controlset or a control master is the eye-gaze itself functioning independentlywithout any necessary need for additional control equipment. Alsomoreover, VRD display is divided into parts, which can provide moreinformation or extra content when eye is gazed to the part of VRDdisplay for a predetermined time period. Time period sets an idle formapping the control according to part of VRD and prevents false gazes orquick gazes, which can be used to discern a scene of VRD.

[0045] Alternatively, a control set or a control master is eye-gazecombined to extra controlling equipment, such as a button, a mouse or akeyboard. After eye-gaze is mapped to VRD, VRD device 504 is in standbymode to utilize eye-gaze based controlling to the device used. Eye-gazeis in a part of VRD display, which user wants to have control, or in apart of VRD display, which is traditionally used to control the device,user confirms to utilize eye-gaze control by implementing an actionusing extra controlling equipment. Action can be a press of button or akeystroke indicating a mark to device to perform control, which happensin response to the user gazing at a button on the display. An exemplaryangle accuracy is ten minutes of arc.

[0046] This paper presents the implementation and embodiments of theinvention with the help of examples. It is obvious to a person skilledin the art, that the invention is not restricted to details of theembodiments presented above, and that the invention can be implementedin another embodiment without deviating from the characteristics of theinvention. Thus, the presented embodiments should be consideredillustrative, but not restricting. Hence, the possibilities ofimplementing and using the invention are only restricted by the enclosedpatent claims. Consequently, the various options of implementing theinvention as determined by the claims, including the equivalentimplementations, also belong to the scope of the present invention.

[0047] For example, the mirror and the reflector have been described ashaving transparencies in order to integrate the device. However, it ispossible to design and arrange the device in such a way, that the lightfrom sources and the picture from the display panel will not meetconcretely and will not disturb one another considerably.

[0048] For another example, the VRD device has been described ascomputer display. However, it is possible to design VRD to smallelectronic devices such as a mobile phone. Embodiments of inventionrequires a simple and light structure to implementation and can beapplied to display of mobile phone. Also low power consumption of thepreferred embodiment of invention is very applicable to mobile devices.

[0049] For another example, after eye-gaze has been mapped on a displayfunctions has been displaying a mark or setting a control. However,various of application can be implemented after mapping eye-gaze. Acontrol can be set to another device, whose part VRD is. E.g.controlling a mechanical or electronic device utilizing embodiments ofinvention which is a part of the entire device.

1. A method of tracking eye-gaze, the method comprising emitting lighthaving a certain wavelength; transferring the light to the retina of aneye; the wavelength of the light being such as to make the fovea of theeye resolvable; detecting light that is reflected from said eye to formdetection information including the resolvable fovea; mapping thedetection information to a predetermined surface, said surface beinglocated at a distance from said eye, the location of the fovea on saidsurface forming an eye-gaze point.
 2. The method of claim 1, furthercomprising using a light having a wavelength in the range of 395 nm to500 nm.
 3. The method of claim 1, further comprising using a lighthaving a wavelength in the range of 395 nm to 430 nm.
 4. The method ofclaim 1, further comprising using a light having a wavelength of 405 nm.5. The method of claim 1, wherein the step of transferring the lightcomprises transferring the light within optics.
 6. The method of claim1, wherein the method comprises narrow band beam splitting at least oneof the emitted and reflected light.
 7. The method of claim 1, whereinthe method rotating the polarization 90 degrees of at least one of theemitted and reflected light.
 8. The method of claim 1, wherein saiddetecting comprises converting said light reflected from said eye retinato electronic information on a detector.
 9. The method of claim 1,wherein said mapping comprises calculating a blood vessels pattern fromsaid detection information; comparing said blood vessels pattern to apredetermined pattern; setting the location of the fovea on said surfacebased on said comparing.
 10. The method of claim 1, and furthercomprising displaying a mark on said predetermined surface at theeye-gaze point.
 11. The method of claim 1, and further comprisingperforming a control function relating to the particular point ofeye-gaze on the surface.
 12. An eye-gaze tracking device, comprising: alight source for emitting light having a certain wavelength; means fortransfering the light to the retina of an eye, the wavelength of thelight being such as to make the fovea of the eye resolvable; a detectorfor detecting light that is reflected from said eye to form detectioninformation including the resolvable fovea; a surface located at adistance from said eye; and means for mapping the detection informationto the surface for locating the the fovea on said surface for forming aneye-gaze point by the location of the fovea.
 13. The device of claim 12,further comprising display as said surface.
 14. The device of claim 12,further comprising a virtual reality display comprising a LCD panel. 15.The device of claim 12, further comprising imaging optics placed betweensaid eye and said surface.
 16. The device of claim 12, furthercomprising a control unit for transferring the detection informationfrom said detector and for converting the detection information to saidpredetermined surface.
 17. The device of claim 12, and comprising anarrow band reflector, a beam splitter and a narrow band mirror forreflecting and filtering the light at a narrow band in the range of saidwavelength.
 18. The device of claim 12, wherein said detector is a CCD(Charge Coupled Device) detector.
 19. The device of claim 12, whereinsaid light source is for emitting a light having a wavelength in therange of 395 nm to 500 nm.
 20. The device of claim 12, wherein saidlight source is for emitting a light having a wavelength in the range of395 nm to 430 nm.
 21. The device of claim 12, wherein said light sourceis for emitting a light having a wavelength of 405 nm.